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REANALYSIS  OF  VALIDATION  OF  TOOL  TO  ASSESS  READINESS  FOR  ONLINE 
LEARNING 

EXECUTIVE  SUMMARY 


Research  Requirement: 

Using  contractors,  the  Army  Continuing  Education  System  (ACES)  developed  an 
assessment  tool  to  determine  Soldiers’  readiness  for  online  learning.  A  U.S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  review  of  the  tool’s  evaluation  report 
recommended  that  the  validation  results  be  re-analyzed  using  correlation-based  approaches.  This 
report  describes  the  reanalysis  of  the  original  web-based  eArmyU  PReP  Questionnaire  (EAPQ) 
concurrent  validation  data  and  its  results.  In  addition,  it  recommends  (a)  changes  to  the 
instrument  (e.g.,  to  the  items  selected  for  inclusion  in  the  final  measure)  to  improve  its 
psychometric  properties  and  (b)  next  steps  for  analysis  of  EAPQ  data  currently  being  collected. 

Procedure: 

After  reviewing  EAPQ  items  and  the  data  base,  statistical  techniques  were  used  to  group 
similar  EAPQ  items  together  to  create  new  proposed  scales  that  were  independent  of  each  other. 
The  psychometric  characteristics  of  the  new  scales  were  evaluated  to  assess  the  extent  to  which 
they  (a)  measure  unique  human  characteristics  reliably  and  (b)  relate  to  each  other  and  student 
demographic  information  in  a  way  that  makes  theoretical  sense.  Another  set  of  statistical 
procedures  was  used  to  assess  the  extent  to  which  these  proposed  scales  are  predictive  of  student 
pace  in  eArmyU  coursework.  Students  varied  on  pace  from  four  courses  behind  to  four  courses 
ahead  of  this  standard.  A  final  set  of  analyses  assessed  whether  the  scales’  capacity  to  predict 
student  pace  was  affected  by  gender  or  race/ethnic  status. 

Findings: 

The  original  analyses  of  the  EAPQ  developed  12  scales  using  90  items.  The  current 
analyses  developed  five  independent  scales  using  53  of  the  90  items.  The  scale  development 
process  revealed  some  concerns  about  some  items  and  the  data  base.  Ten  items  were  dropped 
because  they  depend  on  the  respondent  already  being  enrolled  in  the  eArmyU  and  this  is  not 
consistent  with  the  EAPQ’s  intended  -  preenrollment  -  use.  Additionally,  some  of  the  items 
showed  statistical  relationships  with  each  other  that  do  not  make  theoretical  sense.  This  suggests 
the  possibility  that  some  of  the  items  are  not  labeled  correctly  in  the  data  base  and  could  explain 
why  a  number  of  the  remaining  items  were  not  assigned  to  the  proposed  scales. 

Despite  these  concerns,  53  items  were  associated  with  five  scales  and  the  scale  scores 
show  relationships  with  each  other  supporting  the  hypothesis  that  they  measure  distinct 
characteristics  that  relate  to  each  other  and  the  relevant  demographic  characteristics  in  legitimate 
ways.  This  result  is  considered  good  support  for  the  construct  validity  of  the  scale  scores.  The 
scales  are  titled  (a)  Motivation/Seif-Determination,  (b)  Intolerance  of  Ambiguity,  (c) 


Performance  Goal  Orientation,  (d)  Need  for  Social  Interaction,  and  (e)  Study 
Skills/Conscientiousness. 

The  criterion-related  validity  evidence  for  the  scales  taken  individually  and  as  a  group  is 
good,  but  not  spectacular.  This  means  that  the  proposed  scale  scores  can  be  considered 
reasonable  predictors  of  a  student’s  capacity  to  stay  on  pace  in  the  eArmyU.  However,  it  is 
important  to  note  that  the  sample  of  2,370  respondents  is  not  representative  of  the  population  of 
43,846  students  who  received  the  questionnaire  with  regard  to  their  status  on  pace.  In  fact,  the 
behind-  and  on-pace  students  are  under-represented  in  this  sample  relative  to  ahead-of-pace 
students.  The  EAPQ’s  estimated  capacity  to  predict  pace  would  likely  have  been  greater  if  the 
sample  was  more  representative  of  the  population. 

Utilization  and  Dissemination  of  Findings: 

This  report’s  results  support  the  idea  that  the  EAPQ  items  are  reasonably  predictive  of 
readiness  for  participation  in  the  eArmyU.  However,  it  is  recommended  that  more  work  be  done 
to  show  that  these  results  generalize  beyond  the  data  set  and  sample  of  respondents  examined 
here.  Similar  analyses  could  be  conducted  using  EAPQ  responses  from  an  independent  sample  of 
Soldiers  currently  enrolled  in  eArmyU.  Such  analyses  would  provide  additional  evidence  to 
support  the  use  of  the  EAPQ  for  helping  Soldiers  determine  their  readiness  for  matriculation  into 
the  eArmyU  program. 
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REANALYSIS  OF  VALIDATION  OF  TOOL  TO  ASSESS 
READINESS  FOR  ONLINE  LEARNING 


Introduction 

Through  a  contract  to  Booz  Allen  Hamilton  (BAH),  the  Army  Continuing  Education 
System  (ACES)  developed  an  assessment  tool  to  determine  Soldiers’  readiness  for  online 
learning.  BAH  subcontracted  that  work  to  a  company  named  ComFit  LLC.  A  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  review  of  the  tool’s  evaluation 
report  recommended  that  the  validation  results  be  re-analyzed  using  correlation-based 
approaches.  This  report  describes  the  reanalysis  of  the  original  web  based  eArmyU  PReP 
Questionnaire  (EAPQ)  concurrent  validation  data  and  its  results.  In  addition,  it  recommends  (a) 
changes  to  the  instrument  (e.g.,  to  the  items  selected  for  inclusion  in  the  final  measure)  to 
improve  its  psychometric  properties  and  (b)  next  steps  for  analysis  of  EAPQ  data  currently  being 
collected. 

Taken  together  the  analyses  and  this  report  have  the  following  goals: 

•  Develop  EAPQ  scale  scores  that  assess  relatively  independent  constructs. 

EAPQ  items  from  all  the  original  scales  were  examined  to  identify  similar  items  that 
group  together,  create  new  scales,  and  evaluate  their  independence.  Two  EAPQ  scales 
(e.g.,  Time  Management  and  Tolerance  for  Ambiguity)  are  independent  from  each 
other  if  scores  on  one  scale  are  not  highly  related  to  scores  on  the  other  scale. 
Independence  means  that  the  two  scales  measure  two  different  human  characteristics. 
However,  if  scores  on  one  scale  (e.g.,  Attitude  Towards  Intelligence:  Fixed  or 
Malleable)  are  very  highly  related  to  scores  on  another  scale  (e.g.,  Locus  of  Control), 
these  scores  are  not  independent.  This  suggests  that  they  probably  measure  the  same 
characteristic,  and  therefore  should  be  combined  into  a  single  scale. 

•  Assess  the  internal  consistency  reliability  of  the  resulting  scores. 

Internal  consistency  reliability  is  a  value  between  0  and  1 .00  that  assesses  the  extent 
to  which  the  items  on  the  scale  are  measuring  the  same  characteristic.  Showing  that  a 
scale  is  measuring  a  single  characteristic  is  generally  viewed  as  the  first  step  towards 
making  sure  that  the  scale  is  assessing  the  characteristic  it  is  designed  to  measure. 

•  Examine  the  construct  validity  of  the  scores. 

Construct  validity  asks  the  big  overall  question:  “Does  the  scale  measure  what  it  is 
supposed  to  measure?”  There  are  a  number  of  ways  to  gather  evidence  to  support  this 
inference.  This  report  assesses  whether  (a)  the  scales  are  independent,  (b)  items  on 
each  scale  measure  a  single  characteristic,  (c)  scales  relate  to  each  other  in  a  way  that 
is  consistent  with  the  literature,  and  (d)  scales  relate  to  the  assessed  demographic 
variables  (e.g.,  level  of  education)  in  ways  that  make  sense. 

•  Estimate  the  criterion-related  predictive  validity  of  the  scores. 

In  this  context,  criterion-related  validity  assesses  the  extent  to  which  the  scales,  either 
taken  individually  and/or  all  together,  predict  scores  on  the  outcome  of  interest  (i.e., 
the  criterion).  In  this  case  the  criterion  is  pace  (i.e.,  progress  relative  to  expectation  in 
the  eArmyU  program).  Essentially,  this  looks  at  whether  higher  scale  scores  suggest 
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higher  pace  scores.  If  so,  then  EAPQ  scales  are  correlated  with  pace  and,  in  turn, 
predict  pace.  If  the  EAPQ  scales  predict  pace  and  have  reasonable  construct  validity, 
then  the  EAPQ  is  probably  a  good  tool  for  assessing  whether  Soldiers  are  prepared  to 
enter  eArmyU. 

•  Make  recommendations  for  future  research. 

Based  on  the  results  of  the  analyses  in  this  report,  we  make  recommendations  for 
analysis  of  a  new  data  set  consisting  of  currently  collected  EAPQ  data.  One 
recommendation  is  to  examine  whether  the  recommended  organization  of  items  into 
scales  generalizes  to  the  current  EAPQ.  The  current  EAPQ  is  a  modified  version  of 
the  questionnaire  analyzed  in  this  report.  Another  is  to  assess  the  criterion-related 
validity  of  the  scales  in  the  population  of  eArmyU  participants. 

Brief  Background 

The  operational  goal  for  the  EAPQ  is  to  help  ACES  counselors  (a)  determine  whether 
Soldiers  are  prepared  to  be  successful  in  the  eArmyU  and  (b)  give  feedback  to  Soldiers  who  need 
additional  preparation.  eArmyU  is  a  portal  to  online  college  level  courses  offered  by  a  consortium  of 
universities.  ACES  counselors  have  other  sources  of  information  available  to  inform  their  decision 
(e.g.,  the  General  Technical  (GT)  composite  from  the  Armed  Services  Vocational  Aptitude  Battery 
(ASVAB),  scores  from  the  Test  for  Adult  Basic  Education  (TABE),  and  grade  point  averages 
(GPAs)  from  past  educational  experiences).  However,  these  and  other  predictors  are  not  used  in  a 
predetermined  way  (e.g.,  a  prescribed  formula  or  algorithm);  rather,  the  counselor  makes  a  holistic 
judgment  about  whether  to  recommend  the  Soldier  for  matriculation  into  the  eArmyU. 

Sample 

These  analyses  examined  the  EAPQ  responses  from  the  2,370  enlisted  Soldiers  who 
participated  in  the  concurrent  validation  effort  between  March  5,  2004  and  June  25,  2004.  The 
EAPQ  was  sent  to  the  population  of  43,846  eArmyU  students.  “Concurrent  validation”  refers  to 
the  fact  that  the  Soldiers/students  who  received  and  responded  to  the  questionnaire  were  enrolled 
in  the  eArmyU  at  the  time. 


Measures 

Demographic  Fariab/es 

The  EAPQ  queried  students  on  1 1  demographic  variables.  They  included  (a)  gender,  (b) 
race/ethnicity,  (c)  age,  (d)  marital  status,  (e)  number  of  children,  (f)  level  of  education,  (g)  pay 
grade,  (h)  years  in  the  Army,  (i)  number  of  computer  number  courses  taken  before  eArmyU,  (j) 
number  of  terms  in  eArmyU,  and  (k)  number  of  hours  completed  in  eArmyU. 

Criterion 

We  created  a  new  criterion  variable,  “Pace,”  that  indicates  the  standing  of  a  student 
relative  to  his/her  expected  progression  in  the  eArmyU  program.  Pace  was  obtained  from  the 
original  categorical  variable  “group  membership,”  which  divided  students  into  nine  groups 
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ranging  from  four  courses  behind  pace  to  four  courses  ahead  of  pace.  As  such,  the  Pace  variable 
has  values  from  -4  to  4.  We  treat  this  variable  as  continuous  based  on  the  assumption  that  it  is 
reasonable  to  view  Pace  as  a  continuous  construct.1 

Predictors 

The  original  validation  analyses  derived  the  following  12  scale  scores  from  90  EAPQ 

items: 


•  Computer  Self-Efficacy 

•  Time  Management 

•  Study  Skills 

•  Metacognitive  Awareness 

•  Motivation/Mastery 

•  Personal  Self-Efficacy 

•  Fixed  Intelligence 

•  Need  for  Cognition 

•  Locus  of  Control 

•  Tolerance  for  Ambiguity 

•  Need  for  Social  Interaction 

•  Adequate  Time  for  Studying.2 

Each  item  was  rated  on  a  5-point  Likert  agreement  scale  (i.e.,  1  =  Disagree  Strongly  to  5  =  Agree 
Strongly.) 


Analyses  and  Results 

This  section  describes  the  analyses  and  presents  their  results.  It  starts  out  with  a  review  of 
the  text  of  each  item  and  of  the  data  set  that  was  used  in  the  original  validation  work  and  is  being 
reanalyzed  in  this  report.  Next,  relevant  demographic  data  that  were  collected  along  with  the 
EAPQ  are  examined,  with  a  particular  emphasis  on  comparing  the  distribution  of  the  Pace 
criterion  in  the  population  of  eArmyU  students  who  received  the  questionnaire  to  its  distribution 
for  the  students  who  responded.  Pace  is  the  only  variable  for  which  we  have  the  population 
values.  This  is  followed  by  a  description  of  the  development  of  new  EAPQ  scales.  Finally,  the 
criterion-related  validity  analyses  are  described  and  their  results  reviewed. 

Peview  of  Items 

Appendix  A  shows  the  full  text  of  all  90  EAPQ  items,  organized  by  their  final  scale 
assignment.  Before  the  statistical  analyses  were  performed,  each  item  was  reviewed  for  its 


1  To  maintain  anonymity,  the  concurrent  validation  participants’  Social  Security  Numbers  (SSNs)  were  not 
requested  or  saved.  This  means  that  other  potential  criteria,  like  eArmyU  grade  point  average  (GPA),  are  not 
accessible  for  this  sample. 

2  Because  SSNs  were  not  collected,  as  noted  above,  scores  on  other  potential  predictors  are  not  accessible  (e.g., 
ASVAB  GT  and  TABE  scores). 
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relevance  to  these  analyses.  This  resulted  in  10  items  being  dropped  from  further  analysis.  The 
first  six  items  are  pairs;  one  of  each  pair  asks  about  the  student’s  status  before  beginning 
eArmyU  while  the  other  asks  about  the  student’s  status  now  (after  the  student  has  been  in 
eArmyU  for  some  time).  Given  the  intended  (preenrollment)  use  of  this  instrument,  items  asking 
about  the  student’s  status  during  participation  in  eArmyU  were  eliminated  from  further  analysis 
(i.e.,  items  2, 4,  and  6).  Likewise,  items  9,  10,  and  86  through  90  were  eliminated  from  further 
analysis  because  they  depend  too  much  on  the  student  being  enrolled  in  eArmyU  (e.g.,  Item  9: 
“My  lack  of  computer  skills  or  Internet  knowledge  often  interferes  with  my  ability  to  do  my  best 
work  in  the  eArmyU  courses  I  take”).  Items  11, 12,  44,  and  85  refer  to  the  eArmyU  and  need  to 
be  rewritten;  however,  they  were  included  in  further  analyses  because  the  judgment  was  made 
that  this  reference  does  not  adversely  affect  their  relevance  in  this  context. 

Review  of Data  Set 

The  data  set  we  reviewed  had  2,370  useable  cases.  It  had  missing  data  for  items  86 
through  90  for  all  but  a  few  participants.  This  was  not  a  problem  for  the  analyses  because,  as 
indicated  above,  they  were  eliminated  from  further  analysis  for  other  reasons.  However,  this 
caused  us  to  be  a  little  worried  about  whether  the  responses  for  each  item  were  correctly  matched 
with  the  item/variable  titles  at  the  top  of  each  column.  This  issue  is  discussed  in  more  detail  in 
the  Scale  Development  section  of  this  report. 

Dem  ograp/u'cs 

This  section  presents  the  frequency  results  for  the  demographic  variables,  the  first  of 
which  is  gender  (see  Table  1). 


Table  1.  Distribution  of  Gender 


Gender 

Number 

Percent 

Male 

1,872 

79.0 

Female 

498 

21.0 

Total 

2,370 

100.0 

Table  2  shows  the  distribution  of  race/ethnicity  responses.  Together  Tables  1  and  2  show 
enough  female,  black,  and  Hispanic  participants  to  investigate  gender  and  race/ethnicity 
differences  of  EAPQ  scale  scores  and  conduct  bias  analyses  relative  to  their  prediction  of  the 
Pace  criterion. 


Table  2.  Distribution  of  Race/Ethnic  Subgroups 


Race 

Number 

Percent 

American  Indian  or  Alaska  Native 

30 

1.3 

Asian  or  Pacific  Islander 

50 

2.1 

Black  non-Hispanic 

548 

23.1 

Spanish/Hispanic/Latino 

285 

12.0 

White  non-Hispanic 

1,355 

57.2 

Other 

102 

4.3 

Total 

2,370 

100.0 
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Table  3  shows  the  distribution  of  age  in  this  sample.  The  large  number  of  respondents  31 
years  of  age  or  older  is  interesting. 


Table  3.  Distribution  of  Age 


Age  Range 

Number 

Percent 

20  or  Younger 

72 

3.0 

21  -25 

332 

14.0 

26-30 

412 

17.4 

31  or  Older 

1,554 

65.6 

Total 

2,370 

100.0 

Table  4  shows  the  distribution  of  education  level  across  participants.  A  large  percentage 
of  respondents  have  some  college. 


Table  4.  Distribution  of  Level  of  Education 


Education  Level 

Number 

Percent 

High  school  degree  or  equivalent 

220 

9.3 

Some  college  courses 

1,493 

63.0 

Associate  degree 

514 

21.7 

Bachelor's  degree 

76 

3.2 

Some  graduate  courses 

57 

2.4 

Graduate  degree  or  higher 

10 

0.4 

Total 

2,370 

100.0 

Table  5  shows  the  distribution  of  pay  grades  across  participants.  All  pay  grades  have  some 
representation.  The  distribution  looks  somewhat  normal  with  a  negative  skew.  The  significant 
representation  of  E6  through  E9  Soldiers  is  consistent  with  the  age  distribution  shown  in  Table  3. 


Table  5.  Distribution  of  Pay  Grade 


Pay  Grade 

Number 

Percent 

El 

11 

0.5 

E2 

17 

0.7 

E3 

67 

2.8 

E4 

267 

11.3 

E5 

431 

18.2 

E6 

685 

28.9 

E7 

660 

27.8 

E8 

188 

7.9 

E9 

44 

1.9 

Total 

2,370 

100.0 

Table  6  shows  the  distributions  of  years  in  the  Army.  The  large  percentage  of  responses 
in  the  “Four  or  more”  category  is  consistent  with  results  shown  in  Tables  4  and  5. 
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Table  6.  Distribution  of  Years  in  the  Army 


Years  in  the  Army 

Number 

Percent 

One  or  less 

46 

1.9 

More  than  one  less  than  two 

61 

2.6 

Two  to  three 

134 

5.7 

More  than  three  less  than  four 

93 

3.9 

Four  or  more 

2,036 

85.9 

Total 

2,370 

100.0 

Table  7  shows  the  distribution  of  computer-based  courses  taken  before  entering  the 
eArmyU  program.  This  is  a  somewhat  bimodal  distribution  with  a  large  percentage  of 
participants  (51.3%)  reporting  no  previous  experience  and  the  next  largest  percentage  reporting 
five  classes  or  more  (21 .6%). 

Table  7.  Distribution  of  Computer  Courses  Take  Before  eArmyU 


Number  of  Courses  Taken 

Number 

Percent 

None 

1,215 

51.3 

One 

175 

7.4 

Two 

215 

9.1 

Three 

153 

6.5 

Four 

99 

4.2 

Five  or  more 

513 

21.6 

Total 

2,370 

100.0 

Table  8  shows  the  distribution  of  terms  enrolled  in  the  eArmyU  and  Table  9  shows  the 
distribution  of  number  of  course  hours  completed  in  the  eArmyU.  These  tables  show  that  most  of 
the  sample  had  been  enrolled  between  two  and  five  terms  and  that  the  distribution  of  hours 
completed  was  bimodal  with  a  number  of  Soldiers  at  the  bottom  (i.e.,  0-3  hours)  and  top  (i.e., 
More  than  12). 


Number  of  Terms  in  eArmyU 

Number  Percent 

One 

620  26.2 

Two  to  five 

1,234  52.1 

More  than  five 

516  21.8 

Total 

2,370  100.0 

Table  9.  Distribution  of  Number  of  Hours  in  eArmyU 

Number  of  Hours 

Number 

Percent 

0-3 

805 

34.0 

4-6 

323 

13.6 

7-9 

156 

6.6 

8-12 

333 

14.1 

More  than  1 2 

753 

31.8 

Total 

2,370 

100.0 
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Frequency  distributions  for  marital  status  and  number  of  children  are  not  reported 
because  these  variables  are  not  relevant  for  this  report  and  they  did  not  reveal  any  aberrant 
results  to  cause  concern. 


Table  10  shows  the  distribution  of  the  Pace  criterion  for  the  population  of  students  who 
received  the  EAPQ  and  the  sample  of  students  who  provided  useable  responses.  This  table  shows 
that  behind-  and  on-pace  students  are  under-represented  in  this  sample  relative  to  ahead-of-pace 
students.  This  sample  distribution  looks  very  much  like  it  suffered  from  indirect  range  restriction 
relative  to  the  population.  In  this  case,  indirect  range  restriction  means  that  a  third  variable  (e.g., 
motivation  to  respond)  positively  correlated  with  Pace  influenced  response  rates.  The  difficulty 
is  that  we  cannot  correct  correlations  between  EAPQ  scale  scores  and  Pace  for  indirect  range 
restriction  because  we  do  not  have  data  on  the  third  variable.  This  range  restriction  means  that  it 
is  likely  that  these  results  will  underestimate  the  relationships  between  the  scales  and  the  Pace 
criterion  in  the  population  of  interest. 


Table  10.  Distribution  of  Pace  for  Population  and  Sample 


Pace 

Population 

Sample 

Percent 
Response  Rate 

Number 

Percent 

Number 

Percent 

-4 

2,267 

5.17 

1 

0.04 

0.04 

-3 

3,879 

8.85 

7 

0.30 

0.18 

-2 

5,089 

11.61 

42 

1.77 

0.83 

-1 

4,577 

10.44 

93 

3.92 

2.03 

0 

10,495 

23.94 

385 

16.24 

3.67 

1 

7,059 

16.10 

448 

18.90 

6.35 

2 

4,461 

10.17 

503 

21.22 

11.28 

3 

2,324 

5.30 

280 

11.81 

12.05 

4 

3,695 

8.43 

611 

25.78 

16.54 

Total 

43,846 

100.00 

2,370 

100.00 

5.41 

Note.  Pace  is  a  variable  that  varies  from  -4  to  4.  -4,  4,  and  0  refers  to  four  courses  behind  pace,  four 
courses  ahead  of  pace,  and  on  pace,  respectively. 


Sca/e  Dev e/op m  ent 

Because  the  items  on  the  EAPQ  were  rationally  selected  from  relatively  diverse  domains, 
the  extent  to  which  the  constructs  underlying  those  measures  are  redundant  is  not  clear. 
Accordingly,  the  approach  for  establishing  relatively  independent  scales  and  examining  their 
construct  validity  involved  two  steps:  (a)  empirically  determining  the  factors  (i.e.,  latent 
constructs)  underlying  the  items,  and  (b)  examining  the  patterns  of  relationships  among  the 
resulting  scales  and  with  other  relevant  variables  (e.g.,  gender,  race/ethnicity,  and  education 
level).  However,  preliminary  to  these  steps,  the  statistics  for  individual  items  were  examined  to 
identity  potential  problems  with  particular  items. 


Individual  Item  Statistics 


Table  1 1  shows  basic  distribution  statistics  for  each  item  (i.e.,  minimum,  maximum, 
mean,  and  standard  deviation).  Note  that  items  2,  4,  6,  9, 10,  and  86  through  90  are  not  listed 
because,  as  indicated  above,  they  were  eliminated  from  further  analyses.  None  of  the  items  show 
out  of  range  responses  or  other  distributional  characteristics  that  cause  concern.  The  last  column 
shows  each  item’s  correlation  with  the  Pace  criterion.  The  bolded  correlations  are  significant  (p 
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<  .01);  this  means  that  they  are  statistically  significantly  different  from  zero.  Some  individual 
items  show  substantial  correlations  with  the  Pace  criterion.  For  example  questions  3,  15,  and  83 
correlate  .23,  .19,  and  .17,  respectively,  with  the  Pace  criterion. 

Table  11.  Item  Statistics  and  Correlations  with  the  Criterion 


Correlation 


Item 

Brief  Description 

Minimum 

Maximum 

Mean 

SD 

with  the 
Criterion 
(Pace) 

1 

strongComputerSkillsBefore 

1 

5 

4.19 

0.91 

0.16 

3 

academicConfidenceBefore 

1 

5 

4.35 

0.87 

0.23 

5 

personalConfidenceBefore 

1 

5 

4.54 

0.77 

0.11 

7 

confusedW  ithBasicF  unctions 

1 

5 

2.59 

1.21 

-0.05 

8 

spendAsMuchTimeSurfingAsPossible 

1 

5 

2.48 

1.39 

-0.04 

11 

typingSkillsBefore 

1 

5 

3.59 

1.27 

0.02 

12 

chatSessionsOftenBefore 

1 

5 

3.73 

1.04 

0.06 

13 

trackScheduleSystematically 

1 

5 

4.05 

1.03 

0.04 

14 

canGaugeScheduleAccurately 

1 

5 

2.63 

1.15 

-0.12 

15 

useToDoList 

1 

5 

3.88 

0.98 

0.19 

16 

canMutiTaskOk 

1 

5 

3.36 

1.15 

0.03 

17 

didAssignmentsOnTimelnSchool 

1 

5 

3.77 

1.04 

0.04 

18 

hardToStickToSchedule 

1 

5 

4.39 

0.71 

0.05 

19 

keepUpWithReadings 

1 

5 

3.63 

1.04 

0.04 

20 

outlineMaterials 

1 

5 

2.56 

1.21 

-0.05 

21 

skimNewMaterial 

1 

5 

3.90 

0.82 

0.06 

22 

revisitConfusingMaterial 

1 

5 

3.89 

0.85 

0.08 

23 

organizeNotes 

1 

5 

3.97 

0.87 

0.12 

24 

procrastinates  tudying 

1 

5 

3.91 

0.85 

0.11 

25 

consider  Alternatives 

1 

5 

4.11 

0.80 

0.06 

26 

setGoals 

1 

5 

3.92 

0.90 

0.13 

27 

awareOlExpectations 

1 

5 

3.85 

0.95 

0.02 

28 

periodicallyReview 

1 

5 

4.17 

0.78 

0.07 

29 

chooseBestSolution 

1 

5 

3.77 

0.90 

0.10 

30 

awareOfStudyStrategies 

1 

5 

3.78 

0.93 

0.06 

31 

createOwnExamples 

1 

5 

3.85 

0.91 

0.07 

32 

goodJudgeOfComprehension 

1 

5 

4.42 

0.74 

0.05 

33 

havePurposeF  orEachS  tudyS  trategy 

1 

5 

4.10 

1 

0.05 

34 

changeStrategiesWhenNotWorking 

1 

5 

4.62 

0.65 

0.08 

35 

breakDownStudyingToSmallSteps 

1 

5 

3.88 

1.17 

0.04 

36 

importantToUnderstandThoroughly 

1 

5 

4.09 

0.85 

0.04 

37 

importantT  oDoBetterThanOthers 

1 

5 

3.63 

1.03 

-0.04 

38 

wantT  oLeamAsMuchAsPossible 

1 

5 

4.53 

0.66 

0.13 

39 

fearOfFailureOftenMotivates 

1 

5 

4.33 

0.77 

0.11 

40 

desireToCompletelyMasterMaterial 

1 

5 

4.11 

0.97 

-0.01 

41 

afraidOfNotUnderstandingEnough 

1 

5 

4.48 

0.85 

0.08 

42 

expectToDoWell 

1 

5 

3.91 

1.19 

-0.04 

43 

believeWillReceiveExcellentGrades 

1 

5 

4.35 

0.74 

0.10 

44 

confidentAboutDoingWellWhenEnrolled 

1 

5 

4.35 

0.76 

0.15 

45 

primaryReasonEnhanceCareerOpsCivilian 

1 

5 

4.03 

0.88 

0.12 

46 

primaryReasonEnhanceCareerOpsMilitary 

1 

5 

4.49 

0.71 

0.08 

47 

certainICanmasterSkills 

1 

5 

4.04 

0.81 

0.07 
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Table  11.  (Continued) 


Item 

Brief  Description 

Minimum 

Maximum 

Mean 

SD 

Correlation 
with  the 
Criterion 
(Pace) 

48 

certainlCanDoCoursework 

1 

5 

4.40 

0.15 

49 

canFigureOutMostDifficult 

1 

5 

4.29 

0.76 

0.08 

canDoAlmostAHIfPersistent 

1 

5 

4.32 

0.78 

0.07 

51 

certainCanRememberlnformation 

1 

5 

2.38 

1.12 

-0.06 

52 

certainCanSelfMotivate 

1 

5 

2.90 

-0.05 

53 

canLeamEvenlfHard 

1 

5 

3.16 

1.12 

-0.06 

54 

canDoHardestlflTry 

1 

5 

■sa 

Esa 

0.03 

55 

ratherDoLessChallenging 

1 

5 

am 

m 

0.05 

56 

preferSmallDailyProj  ects 

1 

5 

2.56 

l.n 

-0.05 

57 

likeT  asksRequireLittleThought 

1 

5 

3.30 

1.05 

-0.02 

58 

enjoyT  asksRequiringNewS  olutions 

1 

5 

mm 

1.11 

-0.05 

59 

preferlmportantTasks 

1 

5 

MM 

1.16 

-0.05 

dontCareHowItW  orks 

1 

5 

1.93 

1.12 

-0.08 

61 

deliberateAboutlssues 

1 

5 

3.44 

0.06 

haveCertainFixedlntelligence 

1 

5 

2.51 

-0.05 

63 

cantChangelntelligenceMuch 

1 

5 

-0.09 

64 

cantlncreaseBasicIntelligence 

1 

5 

3.21 

0.97 

0.00 

65 

usuallyGetWhatIWant 

1 

5 

4.14 

0.93 

0.06 

66 

neverKnowWherelStand 

1 

5 

2.16 

0.99 

-0.08 

67 

giveUpWhenNotEasy 

1 

5 

4.38 

0.78 

0.02 

68 

convinceOtherOfMyWay 

1 

5 

4.61 

0.65 

0.06 

69 

mustMasterOwnFate 

1 

5 

1.78 

-0.05 

70 

neverTryThingsUnsureOf 

1 

5 

3.25 

1.14 

-0.05 

71 

eamMyRespect 

1 

5 

3.24 

1.15 

-0.05 

72 

persistenceHardWorkLeadToSuccess 

1 

5 

0.02 

73 

othersControlMyLife 

1 

5 

4.20 

0.04 

74 

impatientWhenNoClearAnswers 

1 

5 

3.28 

-0.02 

75 

frustratedWhenCantGrip 

1 

5 

3.35 

1.07 

-0.10 

76 

whenUnclearlmpulselsToClarify 

1 

5 

3.16 

1.06 

-0.07 

77 

cantStandLeavingUndone 

1 

5 

2.76 

1.14 

-0.09 

78 

dontPerformWell  W  ithV  agueExpectations 

1 

5 

2.55 

1.17 

-0.13 

79 

preferLeaminglnCommunity 

1 

5 

3.25 

1.16 

-0.09 

importantT  oKno  wlnstructor 

1 

5 

3.78 

mm 

-0.05 

81 

UkeCoursesW  ithGroupW  ork 

1 

5 

4.48 

0.13 

82 

preferGroupToSoloAssignments 

1 

5 

3.47 

1.24 

0.14 

83 

leamBetterDiscussingWithOthers 

1 

5 

3.77 

1.12 

0.17 

84 

enjoyLeamingFromOthers 

1 

5 

1.39 

0.12 

85 

completingeArmyUHighPriority 

1 

5 

3.52 

1.26 

0.05 

Note,  n  =  2,370.  Bolded  correlations  are  significant,  p  <  .01. 


Factor  Analysis 

To  identify  the  factors  underlying  the  EAPQ  items,  exploratory  factor  analysis  (EFA)  and 
confirmatory  factor  analysis  (CFA)  were  used  sequentially.  This  approach  is  helpful  to  develop 
scales  when  the  items  are  generated  based  on  constructs  from  diverse  theoretical  domains  (Le, 
Casillas,  Robbins,  &  Langley,  2005).  First,  we  split  the  sample  into  two  sub-samples.  The 
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exploratory  sample  included  about  two-thirds  of  the  total  sample.  The  confirmatory  (i.e.,  hold¬ 
out)  sample  included  the  remaining  one-third  of  the  total  sample. 

EFA  was  carried  out  using  two-thirds  of  the  total  sample  (the  exploratory  sample,  n  = 
1,551).  Principal  axis  factoring  was  used  for  extraction  and  the  extracted  factors  were  rotated  using 
an  oblique  rotation  method  (i.e.,  oblimin).  Based  on  the  scree-plot  and  factor  interpretability,  a 
five-factor  solution  was  selected.  Table  12  shows  the  loadings  on  each  factor.  The  five  factors, 
account  for  about  34%  of  the  total  variance  in  the  80  items  and  are  tentatively  titled  (a) 
Motivation/Seif-Determination,  (b)  Intolerance  of  Ambiguity,3  (c)  Performance  Goal  Orientation, 
(d)  Need  for  Social  Interaction,  and  (e)  Study  Skills/Conscientiousness.  These  factors  are 
reasonable  when  compared  to  past  research.  A  recently  developed  instrument  designed  to  assess 
student  readiness  for  college  found  factors  very  similar  to  Motivation/Seif-Determination  and 
Study  Skills/Conscientiousness  and  a  factor  somewhat  similar  to  Need  for  Social  interaction  (Le  et 
al.,  2005).  The  Intolerance  for  Ambiguity  items  held  together  fairly  well.  This  construct  has  a 
research  history,  and  has  recently  been  being  shown  to  be  predictive  of  U.S.  Army 
noncommissioned  officer  (NCO)  job  performance  (Putka,  Kilcullen,  &  White,  2004).  The 
Performance  Goal  Orientation  factor  is  a  combination  of  items  generally  related  to  confidence  and 
the  belief  that  cognitive  ability  is  not  changeable  (Bell  &  Kozlowski,  2002;  Kozlowski,  Gully, 
Brown,  Salas,  Smith,  &  Nason,  2001).  Table  13  proposes  definitions  of  each  factor. 

The  bolded  loadings  in  Table  12  indicate  the  factor  to  which  the  item  was  assigned.  In 
most  cases  the  loading  was  .30  or  higher  and  it  made  theoretical  sense.  For  example,  it  makes 
theoretical  sense  that  Item  38  (I  want  to  learn  as  much  as  possible  in  any  course  I  take)  has  a 
relatively  high  loading  (.72)  on  the  Motivation/Seif-Determination  factor  and  relatively  low 
loadings  on  other  factors.  These  were  the  59  items  that  were  included  in  the  subsequent  CFA. 

Twenty-one  items  were  not  assigned  to  a  factor  because  they  (a)  did  not  load 
substantially  on  any  factor  (i.e.,  a  loading  of  .30  or  greater)  or  (b)  the  pattern  of  loadings  did  not 
make  sense  (i.e.,  items  7,  8,  12,  13,  14,  20,  37,  41,  54,  55,  57,  61,  64,  65,  67,  68,  69,  72,  73,  74, 
85).  A  number  of  items  with  substantial  loadings  on  factors  that  do  not  make  theoretical  sense 
cause  some  concern.  For  example,  Item  67  (If  I  do  not  succeed  on  a  task,  I  tend  to  give  up)  had  a 
substantial  positive  loading  on  the  Motivation/Seif-Determination  (.42)  and  a  small  negative 
loading  on  Performance  Goal  Orientation  (-.15).  Theoretically,  this  item  should  have  a  negative 
loading  on  Motivation/Seif-Determination  and/or  a  strong  positive  loading  on  Performance  Goal 
Orientation.  Item  13  (I  have  a  systematic  way  [for  example,  an  appointment  book,  diary, 
electronic  calendar,  or  other  method]  to  keep  track  of  my  daily  and  weekly  schedule)  is  another 
example.  It  had  a  relatively  high  negative  loading  on  Need  for  Social  Interaction  when 
theoretically  it  should  have  a  relatively  high  positive  loading  on  Study  Skills/Conscientiousness. 
This  observation  taken  with  the  missing  data  for  the  last  four  items  suggests  that  the  match 
between  item/variable  titles  and  item  responses  in  the  data  set  might  not  be  100%  accurate. 


3  The  construct  Tolerance  for  Ambiguity  was  renamed  Intolerance  of  Ambiguity  to  be  consistent  with  the  scaled 
direction  of  the  items  assessing  it. 
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Table  12.  Exploratory  Factor  Analysis  Results:  Loadings  of  the  Items  on  the  Factors 


Item 

Item  Brief  Description 

Motivation/ 

Self- 

Determination 

Intolerance 

of 

Ambiguity 

Performance 

Goal 

Orientation 

Need  for 
Social 
interaction 

Study 

Skills/ 

Conscien¬ 

tiousness 

1 

strongComputerSkillsBefore 

0.28 

-0.13 

0.03 

0.16 

-0.17 

3 

academicConfidenceBefore 

037 

-0.15 

0.03 

0.24 

-0.18 

5 

personalConfidenceBefore 

0.43 

-0.10 

-0.01 

0.12 

-0.11 

7 

confused  WithBas  icFunctions 

0.07 

0.08 

0.18 

0.00 

-0.06 

8 

spendAsMuchTimeSurfingAsPossible 

0.06 

0.06 

0.13 

0.06 

-0.04 

11 

typingSkillsBefore 

-0.05 

0.00 

0.01 

0.00 

-0.44 

12 

chatSessionsOftenBefore 

0.09 

-0.11 

-0.02 

0.19 

-0.29 

13 

trackScheduleSystematically 

0.20 

-0.01 

-0.01 

0.10 

-0.20 

14 

canGaugeScheduleAccurately 

-0.02 

0.08 

0.19 

-0.33 

0.16 

15 

useToDoList 

0.23 

-0.01 

-0.05 

0.22 

-0.25 

16 

canMutiTaskOk 

-0.07 

0.16 

.  0.02 

0.05 

-0.53 

17 

didAssignmentsOnTimelnSchool 

0.02 

0.09 

-0.01 

-0.01 

-0.51 

18 

hardToStickToSchedule 

0.39 

-0.01 

-0.09 

-0.11 

-0.31 

19 

keepUpWithReadings 

-0.04 

0.11 

-0.03 

-0.02 

-0.62 

20 

outlineMaterials 

0.03 

-0.09 

0.28 

-0.15 

0.25 

21 

skimNewMaterial 

0.07 

0.00 

-0.03 

-0.07 

-0.53 

22 

revisitConfusingMaterial 

-0.01 

0.00 

-0.07 

-0.03 

-0.68 

•  23 

organizeNotes 

0.25 

-0.04 

-0.01 

0.18 

-0.34 

24 

procrastinateStudying 

0.08 

0.07 

-0.03 

0.05 

-0.57 

25 

considerAltematives 

0.08 

0.03 

-0.02 

-0.06 

-0.65 

26 

setGoals 

-0.01 

-0.04 

-0.02 

0.06 

-0.74 

27 

awareOfExpectations 

0.04 

0.00 

0.04 

-0.02 

-0.57 

28 

periodicallyReview 

0.26 

-0.16 

-0.04 

0.02 

-0.47 

29 

chooseBestSolution 

0.01 

-0.07 

0.03 

0.06 

-0.73 

30 

awareOfStudyStrategies 

0.10 

0.02 

0.03 

-0.05 

-0.52 

31 

createOwnExamples 

0.04 

0.04 

0.03 

-0.04 

-0.59 

32 

goodJudgeOfComprchension 

0.47 

0.14 

-0.11 

-0.09 

-0.21 

33 

havePurposeForEachStudyStrategy 

0.44 

0.05 

0.01 

-0.03 

-0.06 

34 

changeStrategiesWhenNotWorking 

0.58 

0.11 

-0.15 

-0.10 

-0.10 

35 

breakDo  wnStudyi  ngT oSmal  1  Steps 

0.33 

0.12 

0.13 

-0.17 

0.03 

36 

importantToUnderstandThoroughly 

0.45 

0.12 

-0.06 

-0.01 

-0.20 

37 

importantToDoBetterThanOthers 

0.19 

0.17 

0.18 

-0.31 

0.08 

38 

wantToLeamAsMuchAsPossible 

0.72 

-0.03 

-0.07 

0.01 

-0.07 

39 

fearOfFailureOflenMotivates 

0.63 

-0.07 

-0.08 

0.08 

-0.10 

40 

desireToCompletelyMasterMaterial 

0.37 

-0.02 

0.02 

0.13 

-0.12 

41 

afraidOfNotUnderstandingEnough 

0.39 

0.08 

0.00 

-0.04 

-0.03 

42 

expectToDoWell 

0.25 

0.12 

0.04 

-0.04 

0.02 

43 

believeWillReceiveExcellentGrades 

0.66 

-0.03 

-0.02 

0.16 

-0.08 

44 

confidentAboutDoingWellWhenEnrolled 

0.61 

-0.01 

-0.07 

0.20 

-0.06 

45 

primary  ReasonEnhanceCareetOpsCi vi  1  i  an 

0.50 

-0,13 

-0.04 

0.24 

-0.12 

46 

primaryReasonEnhanceCareerOpsMilitary 

0.70 

-0.05 

-0.03 

0.09 

0.00 

47 

certainICanmasterS  kills 

0.45 

0.03 

-0.07 

0.19 

-0.23 

48 

certainICanDoCoursework 

0.66 

-0.04 

-0.09 

0.17 

-0.12 

49 

canFigureOutMostDifFicult 

0.63 

-0.05 

-0.11 

0.16 

-0.10 

50 

canDo  Almost  AlllfPersistent 

0.64 

-0.04 

-0.10 

0.17 

-0.06 

51 

certainCariRememberlnformation 

-0.04 

-0.03 

0.55 

-0.02 

0.11 

52 

certainCanSelfMotivate 

0.07 

-0.04 

0.48 

-0.09 

0.12 

53 

canLeamEvenlfHard 

0.15 

0.00 

0.45 

-0.11 

0.07 

54 

canDoHardestlflTry 

0.21 

0.11 

-0.14 

-0.02 

-0.35 

55 

ratherDoLessChallenging 

0.24 

0.02 

-0.06 

0.03 

-0.21 

56 

preferSmallDailyProjects 

0.00 

-0.06 

0.46 

0.06 

0.11 
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Table  12.  (Continued) 


Item 

Item  Brief  Description 

Motivation/ 

Self- 

Determination 

Intolerance 

of 

Ambiguity 

Performance 

Goal 

Orientation 

Need  for 
Social 
interaction 

Study 

Skills/ 

Conscien¬ 

tiousness 

57 

likeTasksRequireLittleThought 

0.09 

-0.14 

58 

enjoyTasksRequiringNewSolutions 

-0.27 

-0.03 

0.68 

59 

preferlmportantT  asks 

-0.26 

0.01 

0.64 

dontCareHowltWorks 

-0.25 

-0.04 

0.63 

61 

dcliberateAboutlssues 

0.23 

-0.11 

-0.17 

62 

haveCertainFixed  Intelligence 

-0.04 

0.05 

0.30 

63 

cantChangelntelligenceMuch 

-0.16 

0.01 

0.47 

64 

cantlncreaseBasiclntelligence 

0.14 

-0.22 

-0.15 

65 

usuallyGetWhatIWant 

0.28 

-0.01 

-0.05 

66 

neverKnowWherelStand 

-0.11 

0.09 

0.41 

67 

giveUpWhenNotEasy 

0.42 

-0.02 

-0.14 

68 

convinceOtherOfMyWay 

0.51 

0.05 

-0.16 

■MMc 

69 

mustMasterOwnFate 

-0.16 

0.04 

§B8iy«B 

70 

neverTryThingsUnsureOf 

0.16 

-0.04 

0.35 

wi 

71 

eamMyRespect 

0.13 

-0.02 

0.33 

-0.42 

72 

persistenceHardWorkLeadToSuccess 

0.28 

-0.05 

-0.24 

-0.23 

73 

othersControlMyLife 

0.28 

0.00 

-0.07 

-0.18 

74 

impatientWhcnNoClearAnswers 

0.01 

-0.24 

0.01 

75 

frustrated  WhenCantGrip 

0.05 

-0.04 

76 

whenUnclearlmpulselsToClarify 

0.10 

0.56 

0.07 

-0.11 

77 

cantStandLcavingUndone 

-0.06 

0.79 

0.01 

0.14 

78 

dontPerformWellWithVagueExpectations 

-0.13 

0.75 

0.05 

0.10 

79 

preferLeaminglnCommunity 

0.76 

-0.01 

-0.01 

80 

importantToKnowInstructor 

0.12 

0.64 

-0.14 

81 

likeCoursesWithGroupWork 

0.56 

0.11 

-0.03 

0.06 

82 

preferGroupToSoloAssignments 

0.36 

0.63 

0.11 

83 

leamBetterDiscussingWithOthers 

0.43 

-0.02 

0.62 

84 

enjoyLeamingFromOthcrs 

0.23 

0.55 

85 

completingeArmyUHighPriority 

0.19 

0.06 

0.38 

Note,  n  =  1,551.  Bolded  loadings  on  assigned  factors  (in  most  cases,  it  means  loading  of  .30  or  higher). 


Table  13.  Definitions  of  Proposed  Factors 

Title _ Definition _ 

Motivation/  The  extent  to  which  students  are  motivated,  purposeful,  and  self-confident. 

Self-Determination 

Intolerance  of  Ambiguity  The  extent  to  which  students  feel  uneasy  when  dealing  with  uncertain  aspects  of 

their  studies;  students’  inability  to  cope  with  uncertainty. 

Performance  Goal  Orientation  The  extent  to  which  students  (a)  have  confidence  on  their  own  abilities,  (b) 

believe  that  abilities  don’t  change,  and  (c)  are  self-conscious  about  their 
performance  relative  to  others. 

Need  for  Social  Interaction  The  extent  to  which  students  prefer  working  collaboratively  with  others. 

Study  Skills/  The  extent  to  which  students  know  how  to  manage  time  and  other  cognitive 

Conscientiousness _ resources  and  consistently  apply  this  knowledge  to  improve  their  study  outcomes. 
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CFA,  on  59  items,  was  then  conducted  using  the  hold-out  sample  (n  =  819)  in  order  to  (a) 
confirm  the  factor  structure,  and  (b)  re-select  items.  Results  were  inconclusive  about  model  fit: 
While  the  root  mean  square  error  of  approximation  (RMSEA)  and  standardized  root  mean 
squared  residual  (SRMR)  suggest  the  model  has  reasonable  fit  (.065  and  .072,  respectively),  the 
comparative  fit  index  (CFI)  and  goodness  of  fit  index  (GFI)  were  not  good  (.746  and  .739, 
respectively).  All  the  loadings,  however,  are  large  (all  statistically  significant)  and  in  the 
expected  direction.  It  is  possible  the  relatively  poor  fit  was  due  to  correlated  residual  errors 
among  the  items  within  factors.  Taken  together,  this  is  sufficient  evidence  to  retain  the  factor 
structure  and  construct  the  scales  accordingly. 

Table  14  presents  the  results  of  the  CFA,  showing  standardized  parameter  estimates  of 
the  items  on  their  specified  factor.  Based  on  these  results,  we  decided  to  exclude  an  additional 
six  items  (items  35,  42,  53,  62,  70,  and  71)  because  of  the  relative  low  loadings.  Items  53  and  62 
had  loadings  that  were  at  our  .30  threshold;  however,  their  loadings  of  .30  and  .31  were  not 
impressive  given  the  other  items  assigned  to  that  factor.  Appendix  A  shows  all  items  organized 
by  the  proposed  factors. 

Table  14.  Confirmatory  Factor  Analysis  Results:  Estimated  Standardized  Loadings  of  the 
Items  on  Assigned  Factors 


Item 

Brief  Description 

Estimated  Standardized 
Loading  on  Assigned 
Factors 

1 

strongComputerSkillsBefore 

0.44 

3 

academicConfidenceBefore 

0.55 

5 

personalConfidenceBefore 

0.48 

11 

typingSkillsBefore 

0.36 

15 

useToDoList 

0.46 

16 

canMutiTaskOk 

0.45 

17 

didAssignmentsOnTimelnSchool 

0.48 

18 

hardToStickToSchedule 

0.55 

19 

keepUpWithReadings 

0.58 

21 

skimNewMaterial 

0.51 

22 

revisitConfusingMaterial 

0.68 

23 

organizeNotes 

0.53 

24 

procrastinateStudying 

0.67 

25 

consider  Alternatives 

0.63 

26 

setGoals 

0.69 

27 

awareOfExpectations 

0.54 

28 

periodicallyReview 

0.58 

29 

chooseBestSolution 

0.75 

30 

awareOfStudyStrategies 

0.54 

31 

createOwnExamples 

0.61 

32 

goodJudgeOfComprehension 

0.57 

33 

havePurposeForEachStudyStrategy 

0.35 

34 

changeStrategiesWhenNotWorking 

0.58 

35 

breakDownStudyingToSmallSteps 

0.17 

36 

importantT  oUnderstandThoroughly 

0.57 

38 

wantToLeamAsMuchAsPossible 

0.74 

39 

fearOfFailureOftenMotivates 

0.72 
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Table  14.  (Continued) 


Item 

Brief  Description 

Estimated  Standardized 
Loading  on  Assigned 
Factors 

40 

desireToCompletelyMasterMaterial 

0.46 

42 

expectToDoWell 

0.20 

43 

believeWillReceiveExcellentGrades 

0.78 

44 

confidentAboutDoingWellWhenEnrolled 

0.76 

45 

primaryReasonEnhanceCareerOpsCivilian 

0.72 

46 

primaryReasonEnhanceCareerOpsMilitary 

0.71 

47 

certainICanmasterS  kills 

0.69 

48 

certainICanDoCoursework 

0.80 

49 

canFigureOutMostDifficult 

0.81 

50 

canDoAlmostAlllfPersistent 

0.77 

51 

certainCanRememberlnformation 

0.43 

52 

certainCanSelfMotivate 

0.35 

53 

canLeamEvenlfHard 

0.30 

56 

prefers  mallDailyProjects 

0.35 

58 

enjoyTasksRequiringNewSolutions 

0.86 

59 

preferlmportantT  asks 

0.84 

60 

dontCareHowItWorks 

0.81 

62 

haveCertainFixedlntelligence 

0.31 

63 

cantChangelntelligenceMuch 

0.38 

66 

neverKnowWherelStand 

0.36 

70 

neverTryThingsUnsureOf 

0.25 

71 

eamMyRespect 

0.26 

75 

frustratedWhenCantGrip 

0.67 

76 

whenUnclearlmpulselsToClarify 

0.58 

77 

cantS  tandLeavingUndone 

0.90 

78 

dontPerform  Well  With  VagueExpectations 

0.87 

79 

preferLeaminglnCommunity 

0.76 

80 

importantT  oKno  wlnstructor 

0.60 

81 

likeCoursesWithGroupWork 

0.52 

82 

preferGroupToSoloAssignments 

0.88 

83 

leamBetterDiscussingWithOthers 

0.90 

84 

enjoyLeamingFromOthers 

0.65 

Note,  n  =  819.  Bolded  loading  refers  to  items  that  were  excluded  due  to  relatively  low  loading  on  the  assigned 
factor. 

Sca/e  Statistics 

As  shown  in  Table  14,  53  items  were  selected  to  construct  five  scales  representing  each 
of  the  factors  found  in  our  EFA  and  CFA  analyses.  Table  15  shows  scale  statistics.  Inter-scale 
correlations  are  shown  in  Table  16.  The  distributions  and  internal  consistency  reliability 
coefficients  of  the  scales  are  reasonable.  The  estimated  reliabilities  for  Intolerance  of  Ambiguity 
and  Need  for  Achievement  are  especially  good  given  the  relatively  small  number  of  items  on 
which  they  are  based.  In  addition,  correlations  among  the  scale  scores  shown  in  Table  16  are 
supportive  of  the  construct  validity  of  the  scales.  It  makes  theoretical  sense  that  Motivation/Self- 
Determination  and  Study  Skills/Conscientiousness  are  relatively  highly  correlated.  It  also  makes 
sense  that  Motivation/Seif-Determination  and  Study  Skills/Conscientiousness  are  negatively 
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correlated  with  Performance  Goal  Orientation.  This  is  because  individuals  who  are  high  on 
Performance  Goal  Orientation  are  motivated  by  the  outward  appearance  of  their  performance 
(i.e.,  extrinsic  motivation),  not  by  an  underlying  motivation  to  understand  and  learn  (i.e.,  intrinsic 
motivation).  Earlier  research  has  indeed  found  a  similar  pattern  of  relationships  between  these 
constructs  (Colquitt  &  Simmering,  1998;  Rawsthome  &  Elliott,  1999). 


Table  15.  Scale  Statistics 


Scale 

Number 
of  Items 

Minimum 

Maximum 

Mean 

SD 

Internal 
Consistency 
Reliability 
(Coefficient  Alpha) 

Motivation/ 

Self-Determination 

19 

19 

95 

82.03 

9.83 

.92 

Intolerance  of  Ambiguity 

6 

6 

30 

18.85 

5.15 

.86 

Performance  Goal  Orientation 

8 

8 

40 

17.61 

5.49 

.79 

Need  for  Social  Interaction 

3 

3 

15 

10.19 

3.29 

.84 

Study  Skills/ 

Conscientiousness 

17 

17 

85 

65.77 

9.59 

.89 

Note.  The  whole  sample  («  =  2,370)  was  used  for  these  analyses. 


Table  16.  Scale  Intercorrelations 


Scale 

1 

2 

3 

4 

5 

1 

Motivation/ 

Self-Determination 

1.00 

2 

Intolerance  of  Ambiguity 

.08 

1.00 

3 

Performance  Goal  Orientation 

-.30 

.08 

1.00 

4 

Need  for  Social  Interaction 

.39 

.04 

-.08 

1.00 

5 

Study  Skills/  Conscientiousness 

.67 

.17 

-.23 

.30 

1.00 

Note,  n  =  2,370.  Bolded  correlations  are  significant,/?  <  .01. 


Tables  17  and  18  show  subgroup  differences  for  the  scale  scores.  There  are  no  substantial 
gender  differences.  However,  there  were  some  noticeable  race/ethnicity  differences.  Both  Blacks 
and  Hispanics  scored  higher  as  a  group  on  Intolerance  for  Ambiguity  than  Whites.  With  regard 
to  Black/White  differences,  this  result  is  consistent  with  past  research  (e.g.,  Putka,  Kilcullen,  & 
White,  2004).  However,  Blacks  and  Hispanics  also  scored  higher  than  Whites  on  Study 
Skills/Conscientiousness.  In  addition,  to  a  lesser  degree  Blacks  scored  higher  than  Whites  on 
Need  for  Social  Interaction. 
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Table  1 7.  Gender  Differences 


Scale 

Male 

Female 

Effect  Size 

Mean 

SD 

Mean 

SD 

Motivation/  Self-Determination 

82.19 

9.77 

81.46 

10.06 

-0.07 

Intolerance  of  Ambiguity 

18.69 

5.14 

19.46 

5.15 

0.15 

Performance  Goal  Orientation 

17.55 

5.39 

17.82 

5.83 

0.05 

Need  for  Social  Interaction 

10.19 

3.26 

10.18 

3.40 

0.00 

Study  Skills/  Conscientiousness 

65.60 

9.60 

66.41 

9.50 

0.08 

Note.  nMa\ee  -  1,872.  nFemate  =  498.  Standard  effect  size  =  (Mfcraaie  -  Mma,c)/SDAn.  Bolded  values  are  significant,/)  < 
.01. 


Table  18.  Race/Ethnicity  Differences 


Scale 

White 

Black 

Hispanic 

Effect  Size 

Mean 

SD 

Mean 

SD 

Mean 

SD 

B/W 

HAV 

Motivation/ 

Self-Determination 

81.83 

9.95 

82.96 

8.92 

82.42 

9.55 

0.11 

0.06 

Intolerance  of  Ambiguity 

17.94 

5.04 

20.52 

4.90 

20.11 

4.85 

0.50 

0.42 

Performance  Goal 

Orientation 

17.36 

5.31 

17.95 

5.84 

17.99 

5.46 

0.11 

0.11 

Need  for  Social  Interaction 

10.18 

3.14 

10.69 

3.38 

9.69 

3.57 

0.16 

-0.15 

Study  Skills/ 
Conscientiousness 

65.05 

9.49 

67.17 

8.98 

67.22 

9.88 

0.22 

0.23 

Note,  nm, lte  ~  1 .355.  nBlack  =  548.  nHispmtc  -  285.  BAV  standard  effect  size  between  Black  and  White  =  (MBiack  - 
Mwhiie)^SDAn.  HAV  standard  effect  size  between  Hispanic  and  White  =  (MHispanie  -  Mwhite)/SDAi|.  Bolded  values  are 
significant,/)  <  .01. 

Table  19  shows  the  correlations  between  the  scale  scores  and  other  relevant  demographic 
variables.  While  a  number  of  the  correlations  are  significant,  none  are  particularly  large  or 
meaningful. 


Table  19.  Correlations  Between  the  Scales  and  Demographic  Variables 


Scale 

Age 

Level  of 
Education 

Pay 

Grade 

Years  in  the 
Army 

Other  Computer 
Courses  Taken 

Motivation/ 

Self-Determination 

.11 

.14 

.15 

.11 

.12 

Intolerance  of  Ambiguity 

.01 

.05 

.01 

-.02 

-.03 

Goal/ Achievement  Orientation 

-.07 

-.04 

-.08 

-.03 

-.06 

Need  for  Social  Interaction 

.05 

.07 

.08 

.01 

.04 

Study  Skills/  Conscientiousness 

.11 

.15 

.16 

.10 

.13 

Note,  n  =  2,370.  Bolded  correlations  are  significant,  p  <  .01. 
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Criterion-Jie/ated  Fa/idily 


This  section  examines  the  extent  to  which  the  proposed  EAPQ  scales  taken  individually 
and/or  all  together  predict  scores  on  the  Pace  criterion.  Table  20  shows  the  zero  order 
correlations  between  each  scale  score  and  the  criterion.  All  of  the  correlations  are  significant  and 
in  directions  that  make  theoretical  sense.  For  example,  it  is  consistent  with  the  literature  that 
Intolerance  for  Ambiguity  and  Performance  Goal  Orientation  have  a  negative  relationship  with  a 
criterion  assessing  academic  performance.  All  of  the  correlations  are  less  than  .20,  and  we 
believe  that  this  modest  level  of  correlation  is  reasonable.  Given  the  academic  nature  of  the  Pace 
criterion,  we  anticipate  that  aptitude-related  variables  (e.g.,  general  cognitive  aptitude  and  past 
academic  performance)  would  have  much  higher  relationships  with  our  performance  outcome. 


Table  20.  Zero-Order  Correlations  with  the  Criterion 


Scale 

Criterion:  Number  of 
Courses  Ahead  of  Pace 

Motivation/Seif-Determination 

0.15 

Intolerance  of  Ambiguity 

-0.11 

Goal/ Achievement  Orientation 

-0.10 

Need  for  Social  Interaction 

0.16 

Study  Skills/  Conscientiousness 

0.12 

Note,  n  =  2,370.  All  correlations  are  significant,  p  <  .01. 


Table  21  shows  the  statistics  associated  with  using  all  five  scale  scores  simultaneously  to 
predict  the  Pace  criterion.  The  multiple  R  associated  with  this  regression  is  .234.  The  beta- 
weights  of  scale  scores  provide  a  sample-specific  means  of  optimally  combining  the  scale  scores 
to  predict  the  Pace  criterion.  However,  this  multiple  R  would  be  an  overestimate  of  the  observed 
relationship  based  on  data  from  a  different  sample.  Therefore,  the  multiple  R  was  adjusted  using 
an  appropriate  shrinkage  formula  to  provide  an  unbiased  estimate  of  the  predictability  of  the 
composite  (e.g.,  Rozeboom,  1978).  The  multiple  R  adjusted  for  shrinkage  is  .226.  Given  the 
temperament/motivational  nature  of  these  scales  and  the  potential  effect  of  indirect  range 
restriction  in  the  current  sample  mentioned  earlier  this  is  not  a  bad  level  of  prediction. 

Table  21.  Regression  Analysis  with  All  the  Scales  Predicting  the  Criterion  Concurrently 


Unstandardized 

Coefficients 

B  Std.  Error 

Standardized 

Coefficients 

Beta 

P 

(Constant) 

0.840 

0.353 

.017 

Motivation/Seif-Determination 

0.012 

0.005 

.069 

.016 

Intolerance  of  Ambiguity 

-0.041 

0.007 

-.129 

.000 

Goal/ Achievement  Orientation 

-0.014 

0.006 

-.046 

.029 

Need  for  Social  Interaction 

0.060 

0.011 

.121 

.000 

Study  Skills/Conscientiousness 

0.008 

0.005 

.047 

.083 

Note,  n  =  2,370. 


Given  that  no  zero-order  validities  estimates  (see  Table  20)  dramatically  outperformed 
the  others,  we  did  not  try  regression  analyses  with  subsets  of  scales.  Additionally,  the  zero-order 
validity  estimates  for  individual  items  that  were  excluded  from  the  scales  were  examined  to  see 
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if  any  of  them  could  be  added  to  improve  the  level  of  prediction  (see  Table  11).  However,  most 
of  the  excluded  items  have  low  individual  correlations  with  the  Pace  criterion  and  none  showed 
any  promise  beyond  the  level  of  prediction  demonstrated  by  the  five  scale  scores. 

Differentia/  Prediction 

Differential  prediction/bias  analyses  were  conducted  for  gender,  Black  versus  White 
race/ethnicity,  and  Hispanic  versus  White  race  ethnicity  effects.  Across  all  five  scales  there  were 
no  effects  for  gender.  That  is,  (a)  the  correlation  between  each  scale  and  the  Pace  criterion  was 
not  significantly  affected  by  gender  at  the  p  <  .01  level  and  (b)  performance  on  the  criterion  was 
not  either  under-  or  over-predicted  for  males  or  females  relative  to  the  overall  sample.  The  main 
effect  due  to  the  Black/White  variable  was  statistically  significant  ip  <  .01)  for  Intolerance  of 
Ambiguity  and  Goal/Achievement  Orientation,  suggesting  that  these  scales  may  slightly  under¬ 
estimate  performance  on  the  Pace  criterion  for  Blacks  (or  overestimate  performance  for  Whites). 
However,  the  effect  is  rather  small  (Beta  =  0.090  and  Beta  =  0.056,  respectively).  There  were  no 
other  significant  Black  versus  White  effects  at  the  p  <  .01  level.  Finally,  the  main  effect  due  to 
the  Hispanic/White  variable  was  statistically  significant  (p  <  .01)  for  Motivation/Self- 
Determination  and  Study  Skills/Conscientiousness,  suggesting  that  the  model  may  slightly 
overestimate  performance  for  Hispanics.  However,  these  effects  were  rather  small  (Beta  =  -0.062 
and  Beta  =  -0.073,  respectively).  There  were  no  other  significant  Hispanic  v.  White  effects  at  the 
p  <  .01  level.  Taken  together,  there  is  very  little  evidence  of  differential  prediction  for  these 
scales  relative  to  gender  or  the  two  race/ethnicity  comparisons  examined. 

Discussion 

Summary  o/Pesu/ts 

The  factor  analysis  approach  was  able  to  develop  EAPQ  scale  scores  that  assess 
relatively  independent  constructs.  The  EFA  and  CFA  identified  53  items  associated  with  five 
scales,  and  the  scale  scores  show  correlations  among  each  other  supporting  the  hypothesis  that 
they  measure  independent  constructs  (see  Table  16).  These  scales  also  showed  strong  internal 
consistency  reliabilities  as  illustrated  in  Table  15,  supporting  the  hypothesis  that  the  items  in 
each  scale  measure  the  same  characteristic  to  a  reasonable  extent.  In  addition,  the  scales  show 
relationships  with  each  other  and  the  examined  demographic  variables  that  make  theoretical 
sense,  providing  additional  evidence  in  support  of  the  construct  validity  of  the  scales. 

There  is  however,  one  significant  concern.  As  described  in  the  Factor  Analysis  section  of 
this  report,  a  number  of  items  showed  substantial  relationships  with  factors  that  do  not  make 
theoretical  sense.  This  observation,  taken  with  the  missing  data  for  the  last  four  items,  suggests 
that  the  match  between  item/variable  titles  and  item  responses  in  the  data  set  might  not  be 
completely  accurate.  If  this  suspicion  is  true,  analyses  of  a  corrected  data  set  might  show  a 
different  structure.  That  is,  more  items  could  load  substantially  on  factors  that  make  sense  and/or 
a  different  set  of  scale  could  be  identified.  While  this  concern  calls  into  question  whether  all  of 
the  85  analyzed  items  are  on  the  correct  scale,  it  does  not  take  away  from  the  evidence  that  the 
EAPQ  items  are  as  a  group  relatively  predictive  of  the  Pace  Criterion.  Corrections  of  any  item 
labeling  mistakes  would  most  likely  lead  to  better  construct  validity  evidence  and  higher 
estimates  of  criterion-related  validity. 
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As  indicated  in  the  Criterion-Related  Validity  section,  the  validity  evidence  for  the  scales 
taken  individually  and  as  a  group  is  good,  but  not  spectacular.  However,  it  is  important  to  note 
that  with  regard  to  the  distribution  of  values  on  the  Pace  criterion,  the  sample  (n  -  2,370)  is  not 
representative  of  the  population  (N=  43,846).  In  fact,  the  behind-  and  on-pace  students  are 
under-represented  in  this  sample  relative  to  ahead-of-pace  students.  Likely  indirect  range 
restriction  for  the  sample  relative  to  the  population  means  that  criterion-related  validity  evidence 
may  be  underestimating  the  relationship  between  the  EAPQ  scales  and  the  Pace  criterion  in  the 
population  of  interest.  That  is,  if  the  sample  was  more  representative  of  the  population,  the 
criterion-related  validity  estimate  would  likely  have  been  greater. 


Imp/ication for  Current  Use  of  the  Questionnaire 

The  version  of  the  EAPQ  that  is  currently  being  used  has  been  modified  since  the  data 
analyzed  for  this  report  were  collected;  however,  the  similarities  are  sufficient  to  make  some 
inferences  about  the  current  version  of  the  instrument,  clarify  questions  that  still  need  to  be 
answered,  and  support  recommendations  for  future  research. 

•  Taken  as  a  whole,  the  items  are  predictive  of  the  Pace  criterion  and  are  potentially 
helpful  in  deciding  whether  a  Soldier  is  ready  to  start  classes  in  the  eArmyU  program. 

•  The  five  scales  that  were  discovered  here  make  conceptual  sense  and  are  somewhat 
related  to  the  scales  that  are  currently  used;  however,  some  questions  remain: 

-  Would  the  same  scales  emerge  with  the  current  version  of  the  questionnaire? 

-  Would  the  same  scales  emerge  with  a  data  set  that  is  known  to  be  free  of  potential 
problems?  With  a  clean  data  set,  more  than  53  items  might  associate  themselves 
with  scales. 

-  Would  the  same  scales  emerge  when  the  sample  consists  of  individuals  who 
completed  the  EAPQ  when  they  were  eArmyU  applicants  instead  of  eArmyU 
participants? 

•  How  would  the  validity  results  differ  in  a  sample  of  Soldiers  that  was  more 
representative  of  the  population  in  terms  of  Pace?  It  is  very  likely  that  the  results 
would  improve. 

•  Would  the  predictive  power  of  the  items/scales  persist  in  an  operational  context? 
Sometimes  when  people  know  the  test  will  affect  an  outcome  they  value,  they  distort 
their  responses  to  make  themselves  appear  more  positive  (i.e.,  faking  good).  This 
phenomenon  could  reduce  the  capacity  of  the  EAPQ  to  assess  readiness. 

•  How  well  does  the  EAPQ  assess  readiness  in  the  context  of  other  available  sources  of 
information  (e.g.,  ASVAB  scores,  TABE  scores,  and  GPA  in  other  recent  educational 
experiences)? 

•  How  well  does  the  EAPQ  predict  other  important  outcome  variables  (i.e.,  criteria) 
like  eArmyU  GPA  and  eventually  completion  of  a  degree? 
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The  bottom  line  here  is  that  the  results  presented  in  this  report  suggest  that  the  EAPQ  items,  as  a 
group,  are  probably  useful  to  the  processes  of  determining  readiness  for  participation  in  the 
eArmyU  program.  However,  more  research  is  needed. 

Recommendation  /or  Future  Research 

We  suggest  conducting  validity  analyses  using  currently  collected  (i.e.,  operational) 
EAPQ  responses.  That  is,  responses  from  eArmyU  participants  that  took  the  current  version  of 
the  questionnaire  when  they  applied  to  the  eArmyU  program.  These  data  would  optimally  be 
accompanied  by  (a)  other  available  predictor  scores  (e.g.,  ASVAB,  TABE  Scores,  GPA  from 
other  recent  courses),  and  (b)  other  criterion  variables  (e.g.,  eArmyU  GPA  and  degree 
completion).4  Such  an  effort  would  include  the  following  steps: 

•  Factor  analyses  need  to  be  conducted  to  establish  that  the  five  scales  discovered  in 
this  study  generalize  to  the  current  version  of  the  questionnaire  used  in  the 
operational  context. 

•  ACES  counseling  subject  matter  experts  (SMEs)  could  be  consulted  with  to  identify 
all  of  the  sources  of  information  that  are  used  and  how  they  are  used  to  provide 
guidance  to  Soldiers  considering  enrollment  in  the  eArmyU  program. 

•  Criterion-related  validity  analyses  could  be  conducted  to  assess  the  predictive  power 
of  the  EAPQ  in  the  operational  context  and  relative  to  other  potential  sources  of 
information  (i.e.,  ASVAB,  TABE  scores,  GPA  from  recent  course  work). 

•  These  criterion-related  validity  analyses  could  also  assess  the  extent  to  which  the 
EAPQ  can  predict  other  important  outcomes  (e.g.,  eArmyU  GPA  and  degree 
completion). 

Based  on  the  information  derived  from  these  steps,  analysts  could  (a)  confidently  establish 
scales,  (b)  make  definitive  recommendations  about  which  items  to  include  or  drop,  (c)  confirm 
the  operational  validity  of  the  current  version  of  the  questionnaire,  and  (d)  make 
recommendations  about  how  the  EAPQ  could  be  used  in  the  process  of  advising  Soldiers  who 
are  considering  participation  in  the  eArmyU  program.  In  addition,  specific  and  empirically 
defensible  EAPQ  cut  scores  or  benchmarks  could  be  identified  to  help  guidance  counselors  in 
evaluating  a  Soldier’s  readiness  for  online  learning.  Advice  on  the  use  of  the  EAPQ  could  also 
include  detailed  guidance  in  how  to  combine  EAPQ  scores  with  other  sources  of  information 
(e.g.,  TABE  and  ASVAB  scores)  in  the  context  of  other  important  educational  outcomes  (e.g., 
eArmyU  GPA  and  likelihood  of  degree  completion)  in  addition  to  Pace. 


4  Retaining  Social  Security  Numbers  (SSNs)  and/or  other  individual  identifiers  for  these  eArmyU  participants  would 
be  very  helpful  for  obtaining  some  of  these  data  from  available  databases  (e.g.,  ASVAB  scores  from  the  enlisted 
master  file  [EMF]). 
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Appendix  A 

Items  and  Their  Final  Scale  Assignments 


Scale/ 

Item 


Item  Description 


Motivation  /Self-Determination 

1  I  had  strong  computer  skills  prior  to  my  enrolling  in  eArmyU. a 

^  Prior  to  enrolling  in  eArmyU,  I  had  a  lot  of  confidence  in  my  ability  to  use  the  Internet 
to  do  academic-  related  research. a 

Prior  to  enrolling  in  eArmyU  I  had  a  lot  of  confidence  in  my  ability  to  use  the  Internet 
to  perform  personal  tasks,  such  as  purchasing  items  or  making  travel  arrangements. 8 

32  I  am  a  good  judge  of  how  well  I  understand  something. 

33  I  have  a  specific  purpose  for  each  study  strategy  I  use. 

34  I  change  strategies  when  I  fail  to  understand  something  I  am  working  on. 

_  ,  It  is  important  for  me  to  understand  as  thoroughly  as  possible  what  is  taught  in  courses 
30  I  take. 

38  .1  want  to  learn  as  much  as  possible  in  any  course  I  take. 

39  My  fear  of  performing  poorly  in  a  course  is  often  what  motivates  me. 

40  I  desire  to  completely  master  the  material  presented  in  the  courses  I  take. 

43  I  believe  I  will  receive  excellent  grades  in  the  courses  I  take  in  the  future. 

44  I  was  confident  when  I  first  enrolled  in  eArmyU  that  I  would  do  very  well. 8 

^  My  primary  reason  for  taking  eArmyU  courses  is  to  enhance  my  career  opportunities 
once  I  am  out  of  the  Army. 

4g  My  primary  reason  for  taking  eArmyU  courses  is  to  enhance  my  opportunities  for 
promotion  in  my  Army  career. 

47  I'm  certain  I  can  master  the  skills  taught  in  the  courses  I  take. 

I'm  certain  I  can  do  my  coursework,  even  when  there  are  other  interesting  things  I 
could  otherwise  do. 

49  I  can  figure  out  how  to  do  even  the  most  difficult  course  work. 

50  I  can  do  almost  all  the  work  in  my  courses  as  long  as  I  don’t  give  up. 

81  I  like  courses  that  involve  group  work. 

Intolerance  of  Ambiguity 

75  I  get  frustrated  when  I  can’t  get  a  grip  on  something. 

76  When  an  issue  is  unclear,  my  impulse  is  to  clarify  right  away. 

77  I  can’t  stand  to  leave  things  undone. 

78  I  don’t  perform  well  when  there  are  vague  expectations  and  goals. 

I  prefer  learning  in  an  environment  where  I  feel  a  sense  of  community  with  other 
students. 

80  When  I  take  a  course,  it’s  important  to  me  that  I  get  to  know  the  course  instructor. 
Performance  Goal  Orientation 

5 1  I'm  certain  I  can  remember  information  presented  in  my  courses. 


Original 

Scale* 


CSE 

CSE 

CSE 

MCA 

MCA 

MCA 

MAL 

MAL 

MAL 

MAL 

MAL 

MAL 

MAL 

MAL 

SEF 

SEF 

SEF 

SEF 

NSI 

TFA 

TFA 

TFA 

TFA 

NSI 

NSI 

SEF 


A-l 


ftem^  ^tem  description 


^2  I'm  certain  I  can  motivate  myself  to  do  the  work  required  of  me  in  order  to  do  well  in 
my  courses. 

56  I  prefer  to  think  about  small  daily  projects  rather  than  long  term  ones. 

58  I  really  enjoy  tasks  that  involve  coming  up  with  new  solutions  to  problems. 

^  I  would  prefer  a  task  that  is  important,  intellectual  and  difficult  over  a  task  that  is 
somewhat  important  but  does  not  require  much  thought. 

60  It’s  enough  for  me  that  something  gets  the  job  done;  I  don’t  care  how  or  why  it  works. 
63  Your  intelligence  is  something  about  you  that  you  can’t  change  very  much. 

66  I  never  know  where  I  stand  with  other  people. 

Need  for  Social  Interaction 

^2  Given  the  choice,  I  would  prefer  working  on  an  assignment  in  a  group  instead  of 
working  on  it  by  myself. 

83  I  learn  better  when  I  can  discuss  my  ideas  with  others. 

84  When  I  work  in  a  team,  I  enjoy  learning  from  other  team  members. 

Study  Skills  /  Conscientiousness 

1 1  My  basic  typing  skills,  prior  to  my  enrolling  in  eArmyU  were  strong. a 

1  Whenever  I  have  a  lot  of  things  to  do  in  a  day,  I  make  a  “to  do”  list  and  check  off  tasks 
as  I  complete  them. 

j  g  Having  to  attend  to  several  tasks  at  the  same  time  has  no  effect  on  my  ability  to 
perform  these  tasks  effectively. 

1 7  When  I  was  in  high  school  or  college,  I  handed  in  my  assignments  on  time. 

1 8  I  find  it  hard  to  stick  to  a  study  schedule. 

19  I  make  sure  to  keep  up  with  the  weekly  readings  and  assignments  for  the  courses  I 
currently  take  or  have  taken  in  the  past. 

21  Before  I  study  new  course  material  thoroughly,  I  often  skim  it  to  see  how  it  is 
organized. 

22  When  I  become  confused  about  something  I’m  reading  for  a  course,  I  go  back  and  try 
to  figure  it  out. 

23  When  I  am  taking  notes,  I  prioritize  and  organize  the  information  I  am  recording. 

24  When  preparing  for  a  test,  I  prefer  to  wait  until  the  last  minute  to  do  my  studying. 

25  I  consider  several  alternatives  to  a  problem  before  I  attempt  to  solve  it. 

26  I  set  specific  goals  before  I  begin  a  task. 

27  I  am  generally  aware  of  what  my  teachers  expect  me  to  learn. 

28  I  periodically  review  to  help  me  understand  important  relationships  between  concepts. 

29  I  think  of  several  ways  to  solve  a  problem  and  choose  the  best  one. 

30  I  am  aware  of  what  strategies  I  use  when  I  study. 

3 1  I  create  my  own  examples  to  make  information  more  meaningful. 


Original 

Scale* 


TMG 


TMG 

TMG 

TMG 


TMG 


MCA 

MCA 

MCA 

MCA 

MCA 

MCA 

MCA 


A-2 


Scale/ 

Item 


Item  Description 


Original 

Scale* 


Excluded 


2 

4 

6 

7 

8 
9 

10 

12 

13 

14 

20 

35 

37 

41 

42 

53 

54 

55 

57 

61 

62 

64 

65 

67 

68 

69 

70 


I  currently  have  strong  computer  skills. b 

I  have  a  lot  of  confidence  right  now  in  my  ability  to  use  the  Internet  to  do  academic- 
related  research. b 

I  have  a  lot  of  confidence  right  now  in  my  ability  to  use  the  Internet  to  perform 

personal  tasks,  such  as  purchasing  items  or  making  travel  arrangements. b 

I  often  find  myself  getting  confused  when  I’m  carrying  out  basic  computer  functions, 

such  as  downloading  a  file  or  searching  for  a  file  on  my  computer. c 

If  I  could  work  it  into  my  schedule,  I  would  spend  as  much  time  as  I  could  surfing  the 

Internet. 

My  lack  of  computer  skills  or  Internet  knowledge  often  interferes  with  my  ability  to  do 
my  best  work  in  the  eArmyU  courses  I  take. b 

I  often  run  into  technical  problems  (computer  malfunction,  bad  Internet  connection, 
missing  software  programs,  etc.)  that  interfere  with  my  ability  to  do  my  best  work  in 
the  eArmyU  courses  I  take. b 

I  often  took  part  in  chat  sessions  and  other  online  activities  (not  necessarily  related  to 
course  work)  prior  to  my  enrolling  in  eArmyU. c 

I  have  a  systematic  way  (for  example,  an  appointment  book,  diary,  electronic  calendar, 
or  other  method)  to  keep  track  of  my  daily  and  weekly  schedule. c 
I  can  accurately  gauge  the  amount  of  time  the  tasks  I’m  scheduled  to  perform  will 
actually  take. c 

When  I  study  the  readings  for  my  courses,  I  outline  the  material  to  help  me  organize 
my  thoughts. c 

I  try  to  break  studying  and  problem  solving  down  into  smaller  steps. c 
It  is  important  for  me  to  do  well  compared  to  others  in  the  courses  I  take. c 

Sometimes  I’m  afraid  that  I  may  not  understand  what  is  being  taught  in  a  course  as 
thoroughly  as  I’d  like  to  understand  it. c 
I  expect  to  do  well  in  the  courses  I  take  in  the  future. c 
Even  if  the  material  in  a  course  is  hard,  I  can  learn  it. c 
I  can  do  even  the  hardest  work  in  my  courses,  if  I  try. c 

I  would  rather  do  something  that  requires  little  thought  as  opposed  to  something  that  is 
sure  to  challenge  my  thinking  abilities. c 

I  like  tasks  that,  once  I’ve  learned  how  to  do  them,  require  little  thought. c 

I  usually  end  up  deliberating  about  issues  even  when  they  do  not  affect  me  personally. c 

You  have  a  certain  amount  of  intelligence  and  you  really  can’t  do  much  to  change  it. 0 

You  can  learn  new  things,  but  you  can’t  really  increase  your  basic  intelligence. c 

I  usually  get  what  I  want  in  life. c 

If  I  do  not  succeed  on  a  task,  I  tend  to  give  up. c 

I  usually  convince  others  to  do  things  my  way. c 

People  must  be  the  master  of  their  own  fate. c 

I  never  try  anything  that  I  am  not  sure  of. c 


CSE 

CSE 

CSE 

CSE 

CSE 

CSE 

CSE 

CSE 

TMG 

TMG 

SSE 

MCA 

MAE 

MAL 

MAL 

SEF 

SEF 

NFC 

NFC 

NFC 

NFC 

NFC 

LOC 

LOC 

LOC 

LOC 

LOC 
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Scale/ 

Item 

Item  Description 

Original 

Scale* 

71 

I  earn  the  respect  and  honors  I  receive. c 

LOC 

72 

Persistence  and  hard  work  usually  lead  to  success. c 

LOC 

73 

Other  people  usually  control  my  life. c 

LOC 

74 

I  get  impatient  when  there  are  no  clear  answers. c 

TFA 

85 

Completing  my  eArmyU  coursework  is  a  high-priority  task  for  me. c 

ATS 

86 

I  have  enough  time  during  a  typical  week  to  take  part  in  activities  of  my  choice,  apart 
from  my  eArmyU  assignments. b 

ATS 

87 

I  have  enough  time  during  a  typical  week  to  complete  my  eArmyU  assignments. b 

ATS 

88 

My  work  schedule  makes  it  easy  for  me  set  up  a  regular  routine  for  working  on  my 
eArmyU  assignments. b 

When  I  have  the  time  to  work  on  eArmyU  course  work,  it  is  easy  to  find  a  quiet  place 
where  I  can  concentrate  on  my  work. b 

ATS 

89 

ATS 

90 

I  am  often  interrupted  when  I  sit  down  to  work  on  my  eArmyU  course  work. b 

ATS 

Note,  a  =  These  items  are  recommended  to  be  revised  to  be  appropriate  for  Soldiers  who  have  not  attended  the 
eArmyU.  b  =  Items  dropped  from  analyses  because  their  contents  are  inappropriate  for  Soldiers  who  have  not 
attended  the  eArmyU.  c  =  Items  dropped  based  on  results  the  Factor  Analyses.  *CSE=  Computer  Self-Efficacy. 


MCA  =  Meta-cognitive  Awareness.  MAL  =  Motivation  and  Attitude  toward  Learning.  SEF  =  Self-Efficacy.  NSI  = 
Need  for  Social  Interaction.  TFA  =  Tolerance  for  Ambiguity.  NFC  =  Need  for  Cognition/ Attitude  for  Intelligence. 
LOC  =  Locus  of  Control.  TMG  =  Time  Management.  SSE  =  Study  Skills  Efficacy.  ATS  =  Adequate  Time  to  Study. 
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